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AN  ANALOG  MODEL  OF  COMPUTATION  FOR 
THE  ILL-POSED  PROBLEMS  OF  EARLY  VISION 


Tomaso  Poggio  and  Christof  Koch 

A  large  gap  exists  at  present  between  computational  theories  of  vision  and  their  possible 
implementation  in  neural  hardware.  The  model  of  computation  provided  by  the  digital 
computer  is  clearly  unsatisfactory  for  the  neurobiologist,  given  the  increasing  evidence  that 
neurons  are  complex  devices,  very  different  from  simple  digital  switches.  It  is  especially 
difficult  to  imagine  how  networks  of  neurons  may  solve  the  equations  involved  in  vision 
algorithms  in  a  way  similar  to  digital  computers.  In  this  paper,  we  suggest  an  analog  model 
of  computation  in  electrical  or  chemical  networks  for  a  larye  class  of  vision  problems,  that 
maps  more  easily  into  biologically  plausible  mechanisms.  Poggio  and  Torre  f4984)*Tfave 
recently  recognized  that  early  vision  problems  such  as  motion  analysis  -{Hem  and  Schunck, 
(981;  Hildreth,  1 984a, b)?  edge  detection  -fForre-  and  Poggio,  1984)? “surface  interpolation 
TGrimson,  1981;  Terzopoulos  1984), c  shape  from-shading  (Ikeucht  and  Horn,  198T)*a'nd 
stereomatching  can  be  characterized  as  mathematically  ill-posed  problems.iw-U»#-sense  of 
Hadam»fd-f1923fr  III  posed  problems  can  be  "SolvecT^fTtccording  to  regularization  theories, 
by  variational  principles  of  a  specific  type.  A  natural  way  of  implementing  variational 
problems  are  electrical,  chemical  or  neuronal  networks.  We  present  specific  networks  for 
solving  several  low-level  vision  problems,  such  as  the  computation  of  visual  motion  and 
edge  detection. 
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Ill-Posed  Problems  and  Variational  Principles 


In  mathematical  analysis  ill  posed  problems  are  defined  as  such  problems  that  do  not  have  a 
unique  solution  that  depends  continuously  on  the  data.  Early  vision  problems  are  ill-posed, 
typically  because  the  solution  is  either  not  uniquely  constrained  by,  or  does  not  depend 
continuously  on  the  data.  Computation  of  the  velocity  field  and  the  recovery  of  shape 
from  shading  are  examples  of  the  first  type;  edge  detection,  considered  as  the  problem 
of  measuring  spatial  changes  of  intensity  in  an  image,  is  an  example  of  the  second  class. 
Rigoreous  regularization  theories  for  "solving"  ill-posed  problems  have  been  developed 
in  recent  years  (see  especially  Tikhonov  and  Arsenin,  1977;  and  Nashed,  1974).  Ill-posed 
problems  of  the  form 


y  =  Az,  (!) 

where  A  is  a  possibly  nonlinear  operator  on  the  unknown  solution  z,  and  y  is  the  data, 
can  be  regularized  by  introducing  suitable  norms  ||-|j,  usually  quadratic,  and  a  “stabilizing" 
functional  ||/’~||,  with  certain  mathematical  properties.  Its  exact  form  depends  on  the 
physical  constraints  that  are  relevant  to  the  specific  problem.  The  regularized  solution  is 
then  obtained  in  terms  of  a  variational  principle.  For  simplicity,  we  consider  here  two  main 
methods  (the  second  and  third  methods  in  Poggio  and  Torre,  1984): 

(a)  If  the  data  are  exact,  then  the  regularized  solution  z  is  the  function  that  minimizes 

II^H  (2) 


among  the  z  such  that  y  —  Az. 

(b)  If  the  data  are  not  exact,  the  regularized  solution  is  z  that  minimizes 

\\Az  -•■  y||2  +  X||Pz||*  (3) 

where  X  is  a  regularizing  parameter  (see  Poggio  and  Torre,  1984). 

Typically,  A  and  /’  are  linear  operators  and  I'  usually  corresponds  to  a  physically  relevant 
smoothness  constraint.  Poggio  and  Torre  (1984)  have  argued  that  most  early  vision  problems 
can  be  solved  in  terms  of  variational  formulations  of  this  type.  A  few  such  solutions  have 
been  already  obtained.  A  representative  example  is  the  computation  of  the  velocity  field 
(Hildreth,  1984a,b;  see  also  Poggio  and  Torre,  1984;  for  a  second  example  see  Poggio, 
Voorhees  and  Yuille,  1984)'.  In  this  case  the  velocity  V  of  each  point  along  a  curve  in  the 
image  is  underdetermined,  since 

V(,i)  —  v^"(a)T(,i)  +•  v-*-(.s)N(a)  (4) 

(T,  N  are  the  unit  vector  tangent  and  perpendicular  to  the  contour  respectively,  .«  is  the 
arc  length),  and  only  i»"r  (.-«)  can  be  obtained  by  local  measurements,  if  the  curve  is  smooth. 
The  stabilizing  functional  derived  by  physical  considerations  (Hildreth,  1984a, b)  is  obtained 
from  /'V  dV/a-i.  The  regularized  solution  is  found  by 

(a)  minimizing 


subject  to  the  data  V7  r ,  if  the  data  are  exact,  or  by 
(b)  minimizing 


where  X-'  is  a  weighting  factor  expressing  the  reliability  of  the  velocity  measurements*, 
when  the  data  are  affected  by  errors. 


Analog  Networks  for  Solving  Variational  Problems 

As  suggested  by  Terzopoulos  (1984),  analog  networks— chemical,  electrical  or  mechanical- 
are  a  natural  computational  model  for  solving  variational  principles.  We  know  from  physics 
that  the  behavior  of  such  systems,  in  fact  the  behavior  of  any  physical  system,  can 
be  described  using  variational  principles  (MacFarlane,  1970).  In  the  frictionless  world  of 
classical  mechanics  a  system's  state  variables  will  behave  in  such  a  way  as  to  minimize 
the  associated  Lagrangian.  Electrical  network  representations  have  been  constructed  for 
practically  all  of  the  field  equations  of  physics  —  many  of  them  are  equivalent  to  variational 
principles  (for  an  electrical  network  implementation  of  ScliFedinger’s  equation  see  Kron, 
1945).  A  fundamental  reason  for  a  natural  mapping  between  variational  principles  and 
electrical  or  chemical  networks  is  Hamilton's  Least  Action  principle3.  The  class  of  variational 
principles  that  can  be  computed  by  analog  networks  is  dictated  by  Kirchhoff’s  current  and 
voltage  laws  (KCL  and  KVL),  which  simply  represent  conservation  and  continuity  restrictions 
satisfied  by  each  network  component  (appropriate  variables  are  usually  voltage  and  current 
for  electrical  networks  and  affinity,  i.e.  chemical  potential,  and  chemical  turnover  rate  for 
chemical  systems).  KCL  and  KVL  provide  the  unifying  structure  of  network  theory.  A  large 
body  of  theoretical  results  is  available  about  networks  satisfying  them,  including  classical 
thermodynamics  (Oster,  Perelson  and  Katchalsky,  1971).  In  particular,  KCL  and  KVL  imply 
Tellegen’s  theorem*.  For  a  network  containing  only  sources  and  linear  resistors,  Tellegen’s 
theorem  implies  Maxwell's  minimum  heat  theorem:  the  distribution  of  voltages  and  currents 
is  such  that  is  minimizes  the  total  power  dissipated  as  heat1'.  These  results  can  be  extended 
to  nonlinear  circuit  components1'  (MacFarlane,  1970;  Oster  &  Desoer,  1971),  but  in  the 
following  we  will  restrict  ourselves  to  linear  resistors.  Since  the  power  dissipated  as  heat 
by  linear  resistors  is  a  quadratic  functional  of  the  form 

(7) 

* 

where  lk  and  Vk  are  the  current  and  the  voltage  respectively,  and  the  sum  is  taken  over  all 
resistive  processes  rk,  it  follows  that  any  network  consisting  of  linear  resistors  and  voltage 
sources  Kk  has  the  following  associated  quadratic  functional  that  is  minimized 

5>/2-£w<  (8) 

k  i 

where  the  second  sum  includes  all  the  batteries.  The  functional  for  the  network  containing 
current  sources  is  very  similar7. 

It  is  then  easy  to  show  formally  that  electrical  networks  of  linear  resistors  and  batteries 
(or  current  sources)  can  solve  quadratic  minimization  problems  in  a  Hilbert  space".  The 
corresponding  linear  equations  ():  l>  have  a  unique  solution  corresponding  to  the  unique 
solution  of  the  variational  principle.  Interestingly,  the  data  b  can  always  be  represented 
in  terms  of  current  or  voltage  sources.  The  matrix  Q  is  the  symmetric,  real  matrix  of  the 
network  resistances". 

A  simpler  argument  also  shows  that  variational  principles  of  the  type  of  equation  (2) 
and  (3)  can  be  solved  by  linear  networks.  The  Euler  Lagrange  equations  associated  with 


equations  (2)  and  (3)  are  linear  in  c.  In  the  discrete  case,  they  correspond  to  n  linear, 
coupled  algebraic  equations  that  can  be  solved  by  networks  of  passive,  linear  resistors  and 
voltage/current  sources".  An  alternative  implementation,  common  on  analog  computers, 
involves  operational  amplifiers  (Jackson,  1960).  As  pointed  out  by  Terzopoulos  in  the 
context  of  vision  (earlier.  Horn  (1974)  proposed  an  analog  implementation  of  the  lightness 
computation),  a  significant  advantage  of  an  analog  implementation  is  its  extreme  parallelism 
and  speed  of  convergence.  Another  advantage  of  resistance  networks  is  their  robustness 
against  random  errors  in  the  resistance  units  of  the  network  (Terzopoulos,  1984;  Karplus, 
1958).  A  disadvantage  is  the  limited  precision  of  the  analog  signals. 


Figure  1 :  Three  resistive  networks  computing  the  smoothest  velocity  field  solution  of  Hildreth,  1984a. 
The  first  two  networks  correspond  to  the  situation  where  the  constraints  imposed  by  the  data  are  to 
be  satisfied  exactly.  The  equation  for  the  current,  which  corresponds  to  the  desired  vT  in  mesh  »' 
(for  figure  la),  is  given  by  (2 r  +  r,)/.  -  r7i+,  -  i  =  ft,,  where  the  value  of  the  battery  ft,  depends 
on  the  velocity  data  v-1  at  location  The  voltage  at  node  i.  corresponding  to  vj.  for  the  network 
lb,  the  dual  of  network  la,  is  given  by  (2 g  +  y,)K,  -  jK+i  -  yV,-i  —  /,,  where  the  injected  current 
depends  on  the  velocity  data.  Sampling  the  voltage  between  nodes  corresponds  to  linear  interpolation 
between  the  node  values.  Network  1c,  consisting  of  two  interconnected  networks  of  the  type  shown 
in  1b,  solves  the  velocity  field  problem  when  the  data  are  not  exact.  The  equations  for  the  i-th  nodes 
are  (2^„  i  y*, )  1 1 ,  y ,  V"», ,  i  qx  V*,'—  i  + <*,-  7^,  ~  tiXi  and  (2y^  +  y , , )  V^,  9^  1  Oy  , — i  c,  y„ 

However,  unlike  the  two  purely  passive  networks  shown  above,  an  active  element  may  be  required, 
since  the  cross-term  <•,,  relating  the  /  and  the  y  components  of  velocity,  can  be  negative.  Such  a 
negative  resistance  can  be  mimicked  by  operational  amplifiers. 


3 


An  Example:  Circuits  for  the  Velocity  Field  Computation 


We  will  consider  next  some  specific  networks  for  solving  the  optical  flow  computation10. 
The  simpler  case  is  when  the  measurements  of  the  perpendicular  component  of  the  velocity, 
t>,x,  at  n  points  along  the  contours  are  exact.  In  this  case,  the  discretized  Euler-Lagrange 
equations,  corresponding  to  the  regularization  solution,  equation  (5),  are  (Hildreth,  1984a) 

(2  +  «?)«>, T  -  vj+i  -  VJ_ ,  =  dit  (9) 

where  k.  is  the  curvature  of  the  curve  at  location  i,  is  a  function  of  the  data  vx  and  the 
curve  and  v[  is  the  unknown  tangential  component  of  the  velocity  t >,  at  location  i  to  be 
computed.  Figure  la  and  lb  shows  two  simple  networks  that  solve  equation  (9),  where  one 
network  is  the  dual  of  the  other. 
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Figure  2:  Two  examples  of  chemical  networks  solving  the  motion  problem  for  exact  measurements. 
They  are  equivalent,  under  steady-state  conditions,  to  the  electric  circuit  of  figure  1b.  Fig.  2a 
illustrates  a  diffusion-reaction  system  A  substance  A  (the  concentration  of  which  corresponds  to 
the  desired  vT )  diffuses  along  a  cable  while  reacting  with  an  extracellular  substance  S  (first  order 
kinetics).  The  corresponding  On-rate  *,  varies  from  location  to  location.  This  could  bo  achieved  by 
a  differential  concentration  of  an  enzyme  catalysing  the  reaction  or  by  varying  the  properties  of 
the  membrane  where  the  reaction  has  to  take  place.  The  Off-rates  can  be  either  constant  or  vary 
with  location.  The  inputs  are  given  by  the  influxes  of  substance  A.  Figure  2b  shows  an  instance  of 
a  lumped  chemical  network,  n  different,  well-mixed  substances,  interact  with  each  other  and  with 
the  substrate  S.  Assuming  first-order  kinetics,  these  reactions  can  mimick  a  linear  resistance  under 
steady  state  conditions.  The  input  is  given  by  the  influx  Mx  and  the  output  by  the  concentration-  of 
X .  Chemical  networks  of  this  type  are  a  kind  of  computational  soup. 


The  equation  describing  the  i-th  node,  in  the  case  of  figure  1b,  is 


(2  g  +  gi)Vi-gVi+i-9Vi-l  =  li  (10) 

where  F,  is  the  voltage— corresponding  to  the  unknown  vT — and  /,  the  injected  current 
at  node  i— corresponding  to  the  measurements  r,1 .  It  is  important  and  surprising  that  this 
implementation  does  not  require  negative  resistances.  When  the  constraints  are  satisfied 
only  approximately  (equation  (6)),  the  equations  are 

(2  4  -  V'v.+1  -  +  c,  14,  =  d„. 

where  b,  depends  on  the  contour  and  V*,  and  VVt  denote  the  x  and  y  component  of  the 
unknown  velocity  i>,  at  location  The  corresponding  network  is  shown  in  Figure  1c.  The 
resistances  r,,  however,  can  be  either  positive  or  negative,  and  may  therefore  require  active 
components  such  as  operational  amplifiers11.  In  the  limit,  as  the  meshes  of  the  circuit 
become  infinitesimally  small,  the  network  solves  the  continuous  variational  problem,  and 
not  simply  its  discrete  approximation. 

We  have  devised  similar  analog  networks  for  solving  other  variational  problems  arising  from 
regularization  analysis  of  several  early  vision  problems  such  as  edge  detection  (Poggio 
ef  a/.,  1984)  and  surface  interpolation1  *  (Terzopoulos,  1984).  These  networks  are  analog 
solution:>  to  certain  kinds  of  spline  interpolation  and  approximation  problems.  For  instance, 
in  the  case  of  surface  interpolation  the  analog  network  solves  the  biharmonic  equation 
which  is  the  Euler-Lagrange  equation  corresponding  to  the  variational  problem  associated 
with  thin  plate  splines.  The  stabilizing  functionals  used  in  regularization  analysis  of  vision 
problems  typically  lead  to  local  and  limited  connections  between  the  components  of  the 
network. 


Solving  Ill-Posed  Problems  with  Biological  Hardware 


The  mathematical  property  of  ill-posedness  determines  a  common  computational  structure 
for  the  solution  to  specific  problems  in  early  vision  based  on  variational  principles  of 
a  specific  type.  Analog  electrical  networks  are  a  natural  hardware  for  computing  such 
variational  principles.  Because  of  the  well-krown  isomorphism  between  electrical  and 
chemical  networks  (see  for  instance  Busse  and  Hess,  1973  or  Eigen,  1974)  that  derives  from 
the  common  underlying  mathematical  structure,  appropriate  sets  of  chemical  reactions  can 
be  devised,  at  least  in  principle,  to  "simulate"  exactly  the  electrical  circuits.  Fig.  2  shows 
two  chemical  networks  that  are  equivalent  (in  the  steady  state)  to  the  electrical  circuit  of 
fig.  lbu. 

Electrical  and  chemical  systems  of  this  type  therefore  offer  a  computational  model  for  early 
vision  that  is  quite  different  from  the  digital  computer.  Equations  are  "solved"  in  an  implicit 
way.  exploiting  the  physical  constraints  provided  by  Kirchhoffs  laws.  It  is  not  difficult  to 
imagine  how  this  model  of  computation  could  be  extended  to  mixed  electrochemical  systems 
by  the  use  of  transducers,  such  as  chemical  synapses,  that  an  decouple  two  parts  of  a 
system,  similarly  to  operational  amplifiers11. 

Could  neural  hardware  exploit  this  model  of  computation?  Increasing  evidence  shows  that 
electrotonic  potentials  play  a  primary  role  in  many  neurons  (Schmitt,  Dev  and  Smith.  1976) 
and  that  membrane  properties  such  as  resistance,  capacitance  and  equivalent  inductance 
(arising  through  voltage  and  time-dependent  conductances;  see  for  instance  Cole,  1968 
and  Koch.  1984)  may  be  effectively  modulated  by  various  types  of  neurotransmitters,  acting 
over  very  different  time  scales  (Marder,  1984).  Dendrodendritic  synapses  and  electrical 
gap  junctions  serve  to  mediate  graded,  analog  interactions  between  neurons  and  do 
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not  rely  on  all  or  ncne  spikes  (Graubard  and  Calvin,  1976).  Such  an  electrical  mediated 
interaction  between  cells  has  been  exploited  in  a  model  of  the  small  signal  behavior  of  the 
photoreceptor  network  in  the  vertebrate  retina  (Torre  and  Owen,  1983). 

When  implementing  electrical  networks  in  equivalent  neuronal  hardware,  one  can  draw  upon 
a  large  number  of  elementary  circuit  elements1’  (Fig.  3).  Patches  of  neuronal  membrane 
or  cytoplasm  can  be  treated  as  resistance  and  capacitance.  Voltage  sources  can  be 
mimicked  by  synapses  on  dendritic  spines  (Koch  and  Poggio,  1983),  whereas  synapses 
on  large  dendrites  act  as  current  sources.  Chemical  synapses  could  effectively  serve  to 
decouple  different  parts  of  a  network.  Chemical  processes  such  as  the  reactions  associated 
with  postsynaptic  effects  or  with  peptides  could  also  be  thought  as  part  of  a  complex 
electrochemical  network.  Obviously,  the  analogy  cannot  be  taken  too  literally.  It  would  be 
very  surprising  to  find  the  exact  neural  analog  of  the  circuit  of  Fig.  3  somewhere  in  the 
CNS.  We  are  convinced,  however,  that  the  style  of  computation  represented  by  analog 
circuits  represents  a  very  useful  model  for  neural  computations  as  well  as  a  challenge  for 
future  VLSI  circuit  designs. 


Figure  3:  This  schematic  figure  illustrates  an  hypothetical  neuronal  implementation  of  the 
regularization  solution  to  the  motion  problem.  A  dendrite,  acting  both  as  pre-  and  post-synaptic 
element  has  a  membrane  resistance  that  can  vary  with  location.  It  can  implement  under  steady-state 
conditions  the  circuit  1b.  The  inputs — corresponding  to  the  measurements  v1-—  are  given  by  synaptic 
mediated  current  injections,  while  the  output  voltages — corresponding  to  the  desired  »T— are  sampled 
by  dendro-dendritic  synapses.  The  membrane  resistance  can  be  locally  controlled  by  suitable  synaptic 
inputs — corresponding  to  the  curvature  of  the  contour— Irom  additional  synapses  opening  channels 
with  a  reversal  potential  close  to  the  resting  potential  of  the  dendrite.  This  scheme  can  easily 
be  extended  to  the  case  where  the  measurements  of  the  perpendicular  velocities  are  not  exact, 
by  having  a  similar,  second  dendrite  (see  also  figure  1  c).  The  interaction  between  both  dendrites 
takes  place  via  two  reciprocal  chemical  synapses.  If  the  corresponding  cross-term  in  equation 
(11)  is  negative,  the  chemical  synapse  must  be  inverting,  presynaptic  depolarization  leading  to  an 
hyperpolarization. 


The  regularization  approach  is  not  limited  to  early  vision.  Other  sensory  modalities  and 
motor  control  can  be  analyzed  in  a  similar  way  (Poggio  and  Torre,  1984).  For  instance,  the 
network  corresponding  to  the  variational  principle  suggested  by  Hogan  for  controlling  the 
formation  of  voluntary  movements  (Hogan,  1984)  is  topologically  related  to  the  electrical 
network  underlying  one-dimensional  edge  detection. 

The  variational  principles  that  we  have  considered  so  far  for  early  vision  processes  are 


quadratic  and  therefore  lead  to  linear  networks.  This  is  not  always  to  be  expected.  The 
ill-posed  problem  of  combining  several  different  sources  of  surface  information,  for  instance, 
may  lead  to  non  quadratic  regularization  expressions  and  to  corresponding  nonlinear 
networks.  Though  this  is  a  topic  that  still  needs  to  be  explored,  nonlinearities  may  greatly 
expand  the  rather  restricted  universe  of  computations  that  can  be  performed  in  terms  of 
quadratic  minimization  principles  (Hopfield,  1984).  Again,  analog  networks  may  be  used 
to  solve  these  minimization  problems,  that  will  in  general  have  multiple  local  minima 
corresponding  to  the  zeros  of  the  mixed  potential  (Brayton  and  Moser,  1964;  Oster,  Perelson 
and  Katchalsky,  1973).  Schemes  similar  to  annealing  (Metropolis,  Rosenbluth,  Rosenbluth, 
Teller  and  Teller,  1953;  Kirkpatrick,  Gelatt  and  Vecchi,  1983;  Hinton  and  Sejnowski,  1983) 
may  be  easily  implemented  by  appropriate  sources  of  gaussian  noise15.  In  a  chemical 
network  "noise"  may  be  introduced  fn  various,  simple  ways.  Needless  to  say,  a  number 
of  biophysical  mechanisms,  such  as  somatic  and  dendritic  spikes,  interactions  between 
conductance  changes,  voltage  and  time-dependent  conductances  etc.,  are  likely  to  be  used 
by  neurons  and  patches  of  membrane  to  perform  a  variety  of  nonlinear  operations. 
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Footnotes 


[1]  In  I  D  the  problem  of  edge  detection  is  considered  as  a  problem  in  numerical 
differentiation  of  the  light  intensity  /(x).  Numerical  differentiation  can  be  regularized  in  the 
following  way.  The  model  is  j n  —  /(xj.  We  want  to  estimate  /'.  We  chose  a  regularizing 
functional  ||/7||  =  / (/"(x)f  </*,  where  /"  is  the  second  derivative  of  /. 

The  solution  can  be  found  by 

(a)  minimizing 

/  \!"Ux  (1.1) 

subject  to  the  data  yit  or  by 

(b)  minimizing 


£  (s/i  -  f(x<))2  +  x/  [/"(i)]2^* 
and  then  in  either  case  by  differentiating  /  to  obtain  /'. 


(1.2) 


[2]  Regularization  theory  provides  techniques  to  determine  the  best  X  (Tikhonov  and  Arsenin, 
1977;  see  also  Reinsch.  1967).  For  instance,  methods  such  as  the  Generalized  Cross 
Validation  method  (Wahba,  1980)  may  be  used  to  find  the  optimal  X.  In  the  case  of  edge 
detection  X  effectively  corresponds  to  the  scale  of  the  filter  (Poggio  et  al.,  1984). 


[3]  The  main  form  of  Hamilton’s  Postulate  of  least  action  states  that  the  motion  of  a 
dissipation-less  dynamical  system,  free  from  external  disturbance,  from  a  configuration  at 
time  t,  to  another  configuration  at  time  t *,  is  such  that  the  integral  of  its  Lagrangian 
h~T  —  U  is  stationary  on  the  path  followed.  That  is 


Ixlt  —  Extremum. 


If  one  extends  Hamilton's  Postulate  to  dissipative  systems  acting  under  external  forces  it 
can  be  shown  that 


—  h’xtu  mum 


must  hold,  where  W  is  the  virtual  work  done  by  the  dissipative  elements  (e.g.  resistors)  and 
the  sources. 


[4)  Kirchhoff’s  laws  are  physical  restatements  of  the  topological  properties  of  the  dynamic 
space  For  electrical  networks  they  correspond  to  conservation  of  flows  (KCL)  and 
uniqueness  of  potential  (KVL).  Tellegen’s  theorem  capture  the  basic  constraints  provided 
by  KCL  and  KVL.  It  is  one  of  the  most  general  and  powerful  results  of  network  theory  and 
is  independent  of  any  assumption  about  constitutive  relations  or  stationarity. 


Tellegen's  theorem :  If  U  is  the  vector  of  branch  potentials  (a  component  for  each  branch) 
and  J  is  the  vector  of  branch  flows,  then 

UTJ  =  0. 

Thus  the  flow  and  the  potential  variables  are  orthogonal  at  any  instant  in  time. 


[5]  If  one  assumes  the  electric  circuit  to  be  isothermal,  then  dividing  the  heat,  Q,  dissipated 
in  the  resistors  by  the  temperature  7\  we  find  that  Q/T  —  dS/dt  is  also  minimum  (Oster  and 
Desoer,  1971).  The  theorem  of  minimum  entropy  production  expresses  a  kind  of  variational 
property  of  linear  nonequilibrium  systems.  When  given  boundary  conditions  prevent  the 
system  from  reaching  thermodynamic  equilibrium  (i.e.  zero  entropy  production)  the  system 
settles  down  in  the  state  of  "least  dissipation”  (Prigogine,  1967). 


[6]  In  addition  to  Hamilton’s  Postulate,  there  are  two  specialized  variational  principles  for 
networks  composed  entirely  of  sources  and  linear  or  nonlinear  dissipative  elements.  In 
uch  a  network,  the  actual  distribution  of  the  currents  will  be  such  as  to  minimize  the  total 
wntent  G  of  the  system  where  G  is  given  by 


Jo 


Conversely,  the  distribution  of  voltages  will  be  such  as  to  minimize  the  total  co-content  / 
of  the  system  where 


-r*. 

Jo 


If  the  network  considered  contains  only  linear  resistors  and  sources,  then  both  of  the  above 
principles  reduce  to  Maxwell’s  Stationary  Heat  Theorem:  the  distribution  of  voltages  and 
currents  will  be  such  as  to  minimize  the  total  power  dissipated  as  heat  (MacFarlane,  1970). 


[7]  In  a  network  of  linear  resistances  r*  =  \Jgk  and  current  sources  /,•  the  associated 
quadratic  power  functional  to  be  minimized  is  given  by 

X>v2*-£av,, 

k  t 

where  the  second  sum  includes  all  the  current  sources. 


[8]  Assume  a  Hilbert  space  with  an  inner  product  <  •,•>,  which  defines  a  quadratic  norm 
||  ||.  An  ill-posed  problem  can  be  formulated  in  terms  of  norms;  i.e.  minimize 

M*  +  Xp*-6||t  (8.1) 

where  /'  is  any  linear  operator  and  b  is  the  data.  This  is  the  third  regularization  method  in 
Poggio  and  Torre  (1984),  Writing  this  in  terms  of  inner  products,  we  have 


•  t  »t*  *p*  •  *  •/ 


<  Pz,  Pz  >  4-X  <  Az  -  b,Az  -  b  >  . 


(8.2) 


This  is  equal  to 

<  Pz,Pz  >  +X  <  Az,Az  >  -X  <  Az,b  >  -X  <  b,  Az  >  X  <  6, 6  >  (8..‘{) 

Since  the  last  term  is  constant,  it  can  be  disregarded  in  the  minimization.  If  the  adjoints  of 
the  operators  P  and  A  are  denoted  by  P'  and  A *  respectively,  minimizing  this  expression 
is  equivalent  to  minimizing 


<  z,  P  Pz  >  +X  <  z,A  Az  >  — 2X  <  z,A  6  >  .  (8. 4) 

Defining  a  new  operator  Q  by  Q  =  P‘  P  +  \A' A,  we  can  formulate  the  original  variational 
problem  as  the  problem  of  minimizing 


<  z,  Qz  >  t-X’  <  z,  A’b  >  (8.5) 

Toe  first  term  can  be  identified  with  the  total  power  dissipated  in  a  linear  resistive  network, 
while  the  second  term  is  the  voltage-  or  current-source  contribution.  Note  that  Q  is 
automatically  self-adjoint.  If  Q  is  a  linear  operator  satisfying 

m  <  z,  z  >  <  <  z,Qz  >  <  M  <  z,  z  > 

for  all  z  e  II,  with  II  being  a  Hilbert  space,  and  some  Af,m  >  0,  the  vector  2  minimizing 
eq.(8.5)  is  the  unique  solution  of  the  equation  Qz  =  b  (thus  the  energy  inner  product 
<  z,Qz  >  is  H-elliptic  and  bounded,  see  Terzopoulos,  1984).  The  problem  of  minimizing 
the  quadratic  functional  on  a  Hilbert  space  can  be  formulated  as  a  Hilbert  space  minimum 
norm  problem  (Luenberger,  1969). 


[9]  Let  the  Euler-Lagrange  equations  associated  with  a  quadratic  variational  problems  be 
Qz  -  b,  where  the  n  dimensional  vector  ft  is  a  function  of  the  data.  These  equations  can 
be  implemented  in  a  planar  network  containing  only  linear  resistances  and  voltage  sources. 
For  every  variable  zt  one  mesh  is  set  up  (with  the  associated  mesh  current  /,).  The  mesh 
consists  of  a  battery  1C,  and  a  resistance  ri}  for  every  non-zero  entry  in  the  *-th  row 
of  Q.  Moreover,  a  very  simple  auxiliary  circuit  connects  the  i-th  and  the  ,;-th  mesh  via  an 
auxiliary  resistance  The  associated  circuit  current  flows  not  only  through  Rt] ,  but  also 
through  rtJ  of  the  i-th  mesh  and  Tj{  of  the  j-th  mesh.  However,  since  this  scheme  requires 
up  to  n-j2  resistances,  it  seems  only  feasible  if  Q  is  sparse.  Fortunately,  the  operator  P  used 
in  the  variational  problems  of  early  vision  (Pogyio  and  Torre,  1984)  has  a  small  and  limited 
support,  so  that  the  corresponding  matrix  Q  will  have  very  few  non-zero  entries.  Although 
t^is  procedure  will  always  yield  an  electrical  network  with  linear  elements  implementing 
(}:  ft,  its  physical  realization  might  require  negative  resistances  (if  the  corresponding 

term  in  Q  is  negative),  which  need  to  be  mimiked  by  active  elements".  Alternatively,  one 
can  use  the  signal-flow  graph  techniques  of  Mason  and  Coates  for  solving  linear  algebraic 
equations  (Kuo,  1967). 


(10)  S.UIIman  (pers.  comm.)  has  suggested  an  analog  mechanical  spring  model  for  the 
computation  of  the  smoothest  velocity  field.  Minimization  of  the  potential  energy  contained 
in  the  springs  corresponds  to  minimization  of  equation  (5)  directly  in  velocity  space.  It 
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has  proved  so  far  impossible  to  map  this  very  special  model  into  electrical  or  chemical 
networks. 


[11]  Physically  realizable  linear  resistances,  whether  in  electrical  or  in  chemical  systems, 
must  dissipate  energy,  i.e.  they  are  constrained  to  the  upper  right  and  the  lower  left 
quadrant  in  the  l-V  plane  and  can  thus  only  be  positive.  However,  there  are  at  least  three 
different  options  for  implementing  negative  resistances  using  basic  circuit  components,  (i) 
The  positive  and  negative  resistances  can  be  replaced  in  a  purely  resistive  network  by 
inductances  and  capacitances,  with  impedance  iw/,  and  -i/{u>C)  respectively.  The  network 
equations  are  then  formulated  in  terms  of  the  currents  and  voltages  at  the  fixed  frequency 
w.  (ii)  The  negative  resistance  can  be  implemented  by  the  use  of  operational  amplifiers 
or  similar  active  circuit  elements.  A  case  in  point  are  reciprocal  inhibitory  synapses,  like 
those  fcund  between  mitral  and  granule  cells  in  the  mammalian  olfactory  bulb.  Depolarizing 
one  system  leads  to  a  hyperpolarization  in  the  second  system,  (iii)  One  can  try  to  exploit 
the  negative  impedance  regions  in  such  highly  nonlinear  systems  as  the  tunnel  diode  or  a 
Hodgkin-Huxley  like  membrane. 


Figure  4:  Electrical  circuits  implementing  the  regularization  solution  to  edge  detection  (Poggio 
el  al.,  1984;  Torre  and  Poggio,  1984)  X~{-  +  /  =  y.  Sampling  /  at  discrete  points  i,  wo  have 
/.+2  -  l/.n  +  S i/i  -  4A_,  h  fi+2  +  \-'fi  =  X'  'yj.  Fig.  4a  shows  the  topology  of  the  circuit  while  fig. 
4b  shows  an  actual  realization.  The  corresponding  network  equation  is 

g-Vi.2  \-gtVm  (2gt  t-  ‘lg.  +  g)Vi  +  9+ Vi-i  +  g-V i-s  •—  if  /<  is  the  current  being  injected  at  the 
i-th  node.  The  negative  resistances  g-  can  be  implemented  using  active  components.  The  resistive 
circuit  can,  however,  be  replaced  by  inductances  and  capacities  and  the  d.c.  voltages  and  currents 
by  a.c.  voltages  and  currents  of  fixed  frequency". 

[  1 2]  A  regularization  analysis  of  the  edge  detection  problem  leads  to  a  simple  one  dimensional 
network  that  implements  the  variational  equation  (1.1).  The  associated  Euler-Lagrange 
equation  is  the  one-dimensional  biharmonic  equation  j-{  —  0,  where  /  is  known  at  some 
discrete  points,  i.e.  /,  f/l,  where  y,  is  the  light  intensity.  The  corresponding  electrical 
network  interpolates  /  between  the  sampling  points.  When  the  measurements  are  not 
exact,  the  Euler-Lagrange  equation  is  given  by  x£{  -t  /{x,-)ll(x  -  x,)  —  y,-4(*  -  *.)• 

The  corresponding  linear,  electrical  circuit  (fig.  4)  approximates  /.  Since  the  corresponding 
elccrical  circuits  are  in  both  cases  space-invariant,  the  underlying  filters  are  convolution 
filters  (Torre  and  Poggio,  1984;  Poggio  cl  al.,  1984). 


[13]  Two  possible  chemical  and  neuronal  implementations  of  the  smoothest  velocity  field 
computation  in  the  case  of  non-exact  measurements  are  shown  in  fig.  5.  They  correspond 
to  the  electrical  circuit  shown  in  fig.  1c. 


a) 


b) 


Figure  5:  A  chemical  and  a  neuronal  implementation  of  equation  (11).  The  cross  term  cit  which 
can  be  negative,  is  mimicked  by  either  an  appropriate  non-linear  chemical  reaction  between  the  two 
substances  Ai  and  II,  or  by  reciprocal  synapses.  For  the  details  compare  with  fig.  2a  and  3. 


[14]  Synapses  may  implement  a  variety  of  linear  and  nonlinear  1-  and  2-ports.  A  chemical 
synapse  converts  a  presynaptic  voltage  Vpre  into  a  postsynaptic  conductance  change 
(via  a  chemical  process)  and  corresponds  to  a  non-reciprocal  2-port  in  the  sense 
of  Brayton  and  Moser  (1964)  or  Oster  et  al.  (1971).  Its  constitutive  relation  defines  a 
2-dimensional  surface  in  the  four-dimensional  space  spanned  by  Vprei  Iprt}  Vpoat  and  Ipoat * 
The  two  presynaptic  variables,  VpTt  and  lprc  are  related  by  Ohm's  law  and  similarly  for  the 
postsynaptic  variables.  The  relation  between  VpTr  and  Vro,t  is  given  by  f{x),  where  /  is 
in  general  a  nonlinear  function.  A  chemical  synapse  is  non-reciprocal,  since  variation  of 
any  of  the  two  postsynaptic  variables  does  not  affect  the  presynaptic  variables.  Chemical 
synapses  tend  to  decouple  neuronal  systems  from  each  other,  rather  like  operational 
amplifiers,  since  a  postsynaptic  current  can  be  induced  without  any  change  in  presynaptic 
current.  Depending  on  /,  synapses  can  mimick  positive  or  negative  resistances.  Reciprocal 
synapses,  like  those  found  between  mitral  and  granule  ceils  in  the  olfactory  bulb  or 
between  bipolar  and  amacrine  cells  in  the  vertebrate  retina  (Ellias  and  Stevens,  1972), 
may  reintroduce  reciprocity.  Because  ~  }{Vt)  and  V =  </(K|),  where  l/i  and  V>  are  the 
intracellular  voltages  of  the  two  neurons,  l'i  can  be  varied  while  observing  its  effect  on 
IV  Upon  reversing  the  position  of  observer  and  input,  one  expects  to  obtain  symmetrical 
measurements  -  at  least  up  to  small  signals.  Electrical  synapses,  usually  referred  to  as 
gap  (unctions,  provide  a  low-resistance  pathways  between  cells,  thus  approximating  linear 
resistances.  However,  rectifying  gap  junctions  are  known  to  exist,  implementing  the  neuronal 
equivalent  of  diodes.  The  transjunctiona!  resistance  can  be  varied  by  a  variety  of  neuronal 
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parameters,  such  as  intracellular  calcium  concentration,  pH  and  voltage  (Bennett,  1972). 


[15]  The  stochastic  differential  equations  describing  an  electrical  or  a  chemical  system  with 
a  source  of  Gaussian  noise  (for  instance  voltage  or  the  presence  of  a  chemical  reactive 
substance)  can  be  formulated  in  terms  of  Ito  or  Stratonovitch  calculus  (Wax,  1954;  Gibman 
and  Skorohod,  1972),  In  physics  the  equations  are  called  Langevin  equations.  They  can 
be  solved  with  the  Fokker-Planck  or  the  Kolmogorov  method.  A  "solution”  of  a  stochastic 
differential  equation  is  a  characterization  in  terms  of  probability  distributions  of  the  "output" 
process  For  linear  networks,  simpler  correlation  methods  can  also  be  used.  If  the  noise  is 
white  and  Gaussian,  its  spectral  density  is  proportional  to  the  "temperature”  T.  The  output 
is  also  a  Gaussian  process  with  probability  distributions  that  asymptotically  approach  the 
Maxwell-Bcltzmann  distribution. 
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